PARAMETER-EFFICIENT FINE-TUNING

State-offset Tuning: State-based Parameter-Efficient Fine-Tuning for State Space Models (2025.acl-short)

Challenge:	Pre-trained language models have demonstrated commendable performance on various NLP tasks.
Approach:	They propose a Parameter-Efficient Fine-Tuning (PEFT) method that incrementally freezes low-rank matrices during fine-tuning to reduce computation and alleviate over-fitting.
Outcome:	The proposed method achieves state-of-the-art performance with an average improvement of 0.85% on the GLUE benchmark while yielding up to 1.86 improvement as opposed to similar PEFT alternatives.

Copied to clipboard

Wonjun Kang, Kevin Galim, Yuchen Zeng, Minjae Lee, Hyung Il Koo, Nam Ik Cho

Adapters Selector: Cross-domains and Multi-tasks LoRA Modules Integration Usage Method (2025.coling-main)

Challenge:	State Space Models (SSMs) have emerged as efficient alternatives to Transformers, but their application to SSMs remains unexplored.
Approach:	They propose a state-based PEFT method that adjusts state directly instead of using external prompts.
Outcome:	The proposed method is based on state-offset tuning, which directly affects state at every timestep.

Copied to clipboard

Yimin Tian, Bolin Zhang, Zhiying Tu, Dianhui Chu

Time Sensitive Knowledge Editing through Efficient Finetuning (2024.acl-short)

Challenge:	Parameter-Efficient fine-tuning (PEFT) adapts large language models to specific domains by updating only a small portion of the parameters.
Approach:	They propose a framework for better integrating usage of multiple adapters by training a middleman adapter to select the appropriate adapter for inference.
Outcome:	The proposed framework can perform cross-domain multi-tasks effectively through the utilization of a compact model in combination with multiple LoRA modules.

Copied to clipboard

Xiou Ge, Ali Mousavi, Edouard Grave, Armand Joulin, Kun Qian, Benjamin Han, Mostafa Arefiyan, Yunyao Li

MoRE: A Mixture of Low-Rank Experts for Adaptive Multi-Task Learning (2025.findings-acl)

Challenge:	Existing locate-and-edit knowledge editing methods suffer from two limitations: they are infeasible for large scale KE in practice and require long run-time.
Approach:	They propose to use parametric fine-tuning techniques to update obsolete knowledge and induce new knowledge into LLMs.
Outcome:	The proposed methods improve the performance of KE and knowledge update in a temporal dataset with knowledge update and knowledge injection examples.

Copied to clipboard

Dacao Zhang, Kun Zhang, Shimao Chu, Le Wu, Xin Li, Si Wei

Low-Rank Adaptation for Multilingual Summarization: An Empirical Study (2024.findings-naacl)

Challenge:	Recent advances in Large Language Models (LLMs) have revolutionized various domains, offering unprecedented performance across numerous tasks.
Approach:	They propose a new Mixture of Low-Rank Experts (MoRE) for multi-task PEFT to improve performance of LLMs with fewer parameters.
Outcome:	The proposed method improves performance over multiple tasks and no additional inference cost.

Copied to clipboard

Chenxi Whitehouse, Fantine Huot, Jasmijn Bastings, Mostafa Dehghani, Chu-Cheng Lin, Mirella Lapata

Information Extraction & Text Mining

Challenge:	Pre-trained Large Language Models have significantly advanced NLP, but their ever-increasing size poses significant challenges for conventional fine-tuning.
Approach:	They investigate the potential of Low-Rank Adaptation (LoRA) in multilingual summarization, a task that is challenging and relatively unexplored.
Outcome:	The proposed method outperforms full fine-tuning and cross-lingual transfer strategies in multilingual summarization tasks.

TeRA: Vector-based Random Tensor Network for High-Rank Adaptation of Large Language Models (2026.acl-long)

Copied to clipboard

Yuxuan Gu, Wuyang Zhou, Giorgos Iacovides, Danilo Mandic

Layer-wise Importance Matters: Less Memory for Better Performance in Parameter-efficient Fine-tuning of Large Language Models (2024.findings-emnlp)

Challenge:	Low-Rank Adaptation (LoRA) methods have significantly reduced the number of trainable parameters needed in fine-tuning large language models.
Approach:	They propose a vector-based random Tensor network for high-Rank Adaptation method that achieves high-rank weight updates while retaining parameter efficiency.
Outcome:	The proposed method outperforms existing PEFT methods while keeping low-rank parameters.

Copied to clipboard

Kai Yao, Penglei Gao, Lichun Li, Yuan Zhao, Xiaofeng Wang, Wei Wang, Jianke Zhu

SparseFit: Few-shot Prompting with Sparse Fine-tuning for Jointly Generating Predictions and Natural Language Explanations (2024.acl-long)

Challenge:	Parameter-Efficient Fine-Tuning (PEFT) methods have gained popularity for adapting pre-trained Large Language Models (LLMs) to downstream tasks.
Approach:	They propose a method to optimize the importance of full layers with layer-wise importance scoring by leveraging the estimated importance scores.
Outcome:	The proposed method is compatible with PEFT methods that operate on a per-layer basis and achieves better performance.

Copied to clipboard

Jesus Solano, Mardhiyah Sanni, Oana-Maria Camburu, Pasquale Minervini

A Single Linear Layer Yields Task-Adapted Low-Rank Matrices (2024.lrec-main)

Challenge:	Models that generate natural language explanations (NLEs) for their predictions often require large datasets of human-written NLEs at training time, which can be expensive and time-consuming to collect.
Approach:	They propose a sparse few-shot fine-tuning strategy that leverages discrete prompts to jointly generate predictions and NLEs.
Outcome:	The proposed approach compares sparse few-shot fine-tuning with existing parametric fine- tuning techniques on three sizes of the T5 language model and four datasets and produces competitive results for both task performance and NLE quality.

Copied to clipboard

Hwichan Kim, Shota Sasaki, Sho Hoshino, Ukyo Honda

Challenge:	Low-Rank Adaptation (LoRA) is a parameter-efficient fine-tuning method that updates initial weight matrix W0 with a delta matrix W .
Approach:	They propose a method that updates initial weight matrix W0 with a delta matrix W consisting of two low-rank matrices A and B.
Outcome:	The proposed method maintains a performance on par with LoRA despite the fact that the trainable parameters of CondLoRA are fewer than those of LoRA.

GAST: Gradient-aligned Sparse Tuning of Large Language Models with Data-layer Selection (2026.eacl-long)

Copied to clipboard

Kai Yao, Zhenghan Song, Kaixin Wu, Mingjie Zhong, Danzhao Cheng, Zhaorui Tan, Yixin Ji, Penglei Gao

STAR: Constraint LoRA with Dynamic Active Learning for Data-Efficient Fine-Tuning of Large Language Models (2024.findings-acl)

Challenge:	Existing methods focus on layer-selective and data-selectory fine-tuning, but ignore the fact that different data points contribute varying degrees to distinct model layers.
Approach:	They propose a method that performs selective fine-tuning at both data and layer dimensions as integral components of a unified optimization strategy.
Outcome:	Experiments show that the proposed method outperforms baseline methods in terms of performance and performance.

Copied to clipboard

Linhai Zhang, Jialong Wu, Deyu Zhou, Guoqiang Xu

SSMLoRA: Enhancing Low-Rank Adaptation with State Space Model (2025.naacl-long)

Challenge:	Existing studies show that supervised training is still necessary for complex reasoning tasks.
Approach:	They propose a method to integrate uncertainty-based active learning and LoRA to effectively integrate the two methods.
Outcome:	The proposed approach outperforms baseline models on three reasoning tasks.

Copied to clipboard

Jiayang Yu, Yihang Zhang, Bin Wang, Peiqin Lin, YongKang Liu, Shi Feng

Green & Sustainable NLP

Challenge:	Fine-tuning requires substantial computational resources and is prone to overfitting when applied to small datasets.
Approach:	They propose a parameter-efficient fine-tuning method that integrates a State Space Model (SSM) to interconnect low-rank matrices.
Outcome:	The proposed method achieves comparable performance to LoRA on the general language understanding evaluation (GLUE) benchmark while using only half the parameters.

Music for All: Representational Bias and Cross-Cultural Adaptability of Music Generation Models (2025.findings-naacl)

Copied to clipboard

Atharva Mehta, Shivam Chauhan, Amirbek Djanibekov, Atharva Kulkarni, Gus Xia, Monojit Choudhury

SuLoRA: Subspace Low-Rank Adaptation for Parameter-Efficient Fine-Tuning (2025.findings-acl)

Challenge:	Existing music generation models are limited in their coverage of the musical genres and cultures of the world.
Approach:	They propose to use parametric fine tuning techniques to mitigat the bias in existing music datasets.
Outcome:	The proposed models are able to perform well across genres and cultures.

Copied to clipboard

Chenhao Ding, Jiangyang Li, SongLin Dong, Xinyuan Gao, Yuhang He, Yihong Gong

MALoRA: Mixture of Asymmetric Low-Rank Adaptation for Enhanced Multi-Task Learning (2025.findings-naacl)

Challenge:	Existing methods for fine-tuning large language models (LLMs) introduce parameter interference, leading to a gap in generalization performance for specific tasks compared to full fine-uning.
Approach:	They propose a parameter-separated low-rank adapter to account for task differences by decomposing LoRA’s parameter matrix into multiple independent subspaces and assigning them differentially to distinct tasks.
Outcome:	The proposed method outperforms LoRA in trainable parameter efficiency and overall model performance on various NLP tasks.

Copied to clipboard

Xujia Wang, Haiyan Zhao, Shuo Wang, Hanqing Wang, Zhiyuan Liu

PEMA: An Offsite-Tunable Plug-in External Memory Adaptation for Language Models (2024.naacl-long)

Challenge:	Large Language Models (LLMs) can be fine-tuned to new tasks, but in multi-task scenarios, training imbalance and seesaw effect often arise.
Approach:	They propose a flexible fine-tuning framework that leverages asymmetric optimization among LoRA experts to reduce training imbalance and improve performance.
Outcome:	The proposed framework outperforms baseline methods in inter- and intra-task learning scenarios.

Copied to clipboard

HyunJin Kim, Young Jin Kim, JinYeong Bak

LoSiA: Efficient High-Rank Fine-Tuning via Subnet Localization and Optimization (2025.emnlp-main)

Challenge:	Pre-trained language models (PLMs) show impressive performance in various downstream NLP tasks.
Approach:	They propose a Parameter-Efficient Fine-Tuning method that integrates with context representations from test data to perform downstream tasks.
Outcome:	The proposed method outperforms other methods in memory and latency efficiency and maintains sentence meaning and generating appropriate language and styles.

Copied to clipboard

Xujia Wang, Yunjia Qi, Bin Xu

Prompt Compression for Large Language Models: A Survey (2025.naacl-long)

Challenge:	Existing methods perform extensive matrix multiplications in domain specialization tasks, resulting in computational inefficiency and sub-optimal fine-tuning performance.
Approach:	They propose a method that localizes and optimizes critical parameters during training . they propose 'LoSiA-Pro' which reduces training latency by 27% .
Outcome:	The proposed method achieves minimal performance drop compared to full fine-tuning while requiring the least training time across domain specialization and common-sense reasoning tasks.

Copied to clipboard

Zongqian Li, Yinhong Liu, Yixuan Su, Nigel Collier

Hit the Nail on the Head: Parameter-Efficient Multi-task Tuning via Human Language Intervention (2024.findings-emnlp)

Challenge:	Current methods for improving LLM efficiency focus on optimizing the model itself, while prompt-centric methods focus on lowering the complexity of input.
Approach:	They propose to use prompt compression to optimize the compression encoder and combine hard and soft prompt methods to improve the efficiency of LLMs.
Outcome:	The proposed methods are categorized into hard prompt methods and soft prompt methods.

Copied to clipboard

Wenxuan Lu, Songhao Jiang, Wang Yijing, Tianning Zang

Semantic are Beacons: A Semantic Perspective for Unveiling Parameter-Efficient Fine-Tuning in Knowledge Learning (2024.findings-acl)

Challenge:	Recent studies show that PEFT on small pre-trained language models improves multitasking capabilities.
Approach:	They propose a multi-task learning framework that enables transfer of prior knowledge across tasks . they attach task descriptions to input samples and map them to task embeddings .
Outcome:	The proposed method improves performance on a T5 model and in decoder-only models .

Copied to clipboard

Renzhi Wang, Piji Li

An Empirical Study on Parameter-Efficient Fine-Tuning for MultiModal Large Language Models (2024.findings-acl)

Challenge:	Parameter-Efficient Fine-Tuning (PEFT) methods allow efficient adaptation of Large Language Models (LLMs) to various downstream tasks, but their effectiveness diminishes when downstream tasks require accurate learning of specific knowledge.
Approach:	They propose a method that fine-tunes a limited number of model parameters while keeping the majority of original parameters fixed.
Outcome:	The proposed method is able to perform on open-source large language models and validates the semantic challenge in PEFT.

Copied to clipboard

Xiongtao Zhou, Jie He, Yuhua Ke, Guangyao Zhu, Victor Gutierrez Basulto, Jeff Pan

Let’s Focus on Neuron: Neuron-Level Supervised Fine-tuning for Large Language Model (2025.coling-main)

Challenge:	Multimodal Large Language Models fine-tuned with multimodal instruction-following data have demonstrated formidable capabilities in multimodal tasks.
Approach:	They propose to employ four PEFT methods to fine-tune the LLM component of open-source MLLMs.
Outcome:	The proposed method is the best performing on seven datasets, while fine-tuning the connector layers leads to improved performance in most MLLMs.

Copied to clipboard

Haoyun Xu, Runzhe Zhan, Yingpeng Ma, Derek F. Wong, Lidia S. Chao

TeamLoRA: Boosting Low-Rank Adaptation with Expert Collaboration and Competition (2025.acl-long)

Challenge:	Large Language Models (LLMs) are composed of neurons that exhibit diverse behaviors and roles.
Approach:	They propose a novel approach that refines the granularity of parameter training down to the individual neuron, enabling a more parameter-efficient fine-tuning model.
Outcome:	The proposed approach exceeds the performance of full-parameter fine-tuning and PEFT and provides insights into the analysis of neurons.

Copied to clipboard

Tianwei Lin, Jiang Liu, Wenqiao Zhang, Yang Dai, Haoyuan Li, Zhelun Yu, Wanggui He, Juncheng Li, Jiannan Guo, Hao Jiang, Siliang Tang, Yueting Zhuang

Sensitivity-LoRA : Low-Load Sensitivity-Based Fine-Tuning for Large Language Models (2025.findings-emnlp)

Challenge:	Existing methods for fine-tuning are resource-efficient, but performance often falls short . a new approach, TeamLoRA, integrates collaborative and competitive modules to improve performance.
Approach:	They propose to introduce task-specific LoRA as domain experts to improve learning efficiency . teamLoRA integrates collaborative and competition modules to improve model learning .
Outcome:	Experiments show that TeamLoRA improves performance in multi-task learning . teamLorea integrates collaborative and competitive modules to improve performance .

Copied to clipboard

Hao Zhang, Bo Huang, Zhenjia Li, Xi Xiao, Hui Yi Leong, Zumeng Zhang, Xinwei Long, Tianyang Wang, Hao Xu

Know-Adapter: Towards Knowledge-Aware Parameter-Efficient Transfer Learning for Few-shot Named Entity Recognition (2024.lrec-main)

Challenge:	Low-Rank Adaptation (LoRA) is a promising approach to adapting LLMs to specialized tasks . existing rank allocation techniques remain computationally inefficient and unstable .
Approach:	They propose a low-rank adapted model that approximates model weight updates using low-ranked decomposition.
Outcome:	The proposed method is limited by its uniform rank allocation to each incremental matrix . it leverages the second-order derivatives of the loss function to capture weight sensitivity .

Copied to clipboard

Binling Nie, Yiming Shao, Yigang Wang

NormAL LoRA: What is the perfect size? (2025.findings-emnlp)

Challenge:	Named entity recognition (NER) is a fundamental task in natural language processing.
Approach:	They propose a knowledgeable adapter to incorporate structure and semantic knowledge of knowledge graphs into PLMs for few-shot NER.
Outcome:	The proposed adapter improves the quality of retrieved information by adding explicit knowledge from external sources to PEFTs.

Copied to clipboard

null Aastik, Topu Sai Meghana, Chinmay Prakash Kulkarni, Pragya Paramita Sahu

Green & Sustainable NLP

Challenge:	Large Language Models (LLMs) are crucial for enabling intelligent experiences across applications.
Approach:	They propose a low-rank adaptive localization method that uses rank-norm regularization to determine the optimal rank for each weight matrix.
Outcome:	NormAL LoRA reduces adapter parameters by 37% while preserving full fine-tuning performance.

GPT vs RETRO: Exploring the Intersection of Retrieval and Parameter-Efficient Fine-Tuning (2024.emnlp-main)

Copied to clipboard

Aleksander Ficek, Jiaqi Zeng, Oleksii Kuchaiev

Enhancing Chain-of-Thought Reasoning with Critical Representation Fine-tuning (2025.acl-long)

Challenge:	Pre-trained large language models can be used for specific tasks and unique information but lack the resources for extensive retraining.
Approach:	They propose to use PEFT methods to adapt large language models while minimizing compute requirements.
Outcome:	The proposed methods outperform GPT models in zero-shot settings but lag behind PEFT.

Copied to clipboard

Chenxi Huang, Shaotian Yan, Liang Xie, Binbin Lin, Sinan Fan, Yue Xin, Deng Cai, Chen Shen, Jieping Ye

Language Models Can be Efficiently Steered via Minimal Embedding Layer Transformations (2025.emnlp-main)

Challenge:	Representation Fine-tuning (ReFT) is a proposed method for improving parameter efficiency . however, it yields suboptimal performance, as fixed-position representations have uncertain impact on outputs .
Approach:	They propose a method that fine-tunes critical representations in a low-rank linear subspace while freezing the base model.
Outcome:	The proposed method improves accuracy of LLaMA-2-7B and ReFT by 18.2 and 3.8 on GSM8K.

Copied to clipboard

Diogo Tavares, David Semedo, Alexander Rudnicky, Joao Magalhaes

GSQ-Tuning: Group-Shared Exponents Integer in Fully Quantized Training for LLMs On-Device Fine-tuning (2025.findings-acl)

Challenge:	Existing methods for fine-tuning Large Language Models (LLMs) neglect the embedding layer.
Approach:	They propose a PEFT approach that modifies input embeddings without altering hidden layers.
Outcome:	Experiments show that TinyTE modifies embeddings without altering hidden layers . the proposed approach achieves competitive performance while requiring 0.0001% of parameters .

Copied to clipboard

Sifan Zhou, Shuo Wang, Zhihang Yuan, Mingjia Shi, Yuzhang Shang, Dawei Yang

Refining Salience-Aware Sparse Fine-Tuning Strategies for Language Models (2025.acl-long)

Challenge:	Large Language Models (LLMs) fine-tuning techniques require large Floating Point(FP) computation and are impractical for resource-constrained edge devices.
Approach:	They propose a framework for on-device LLM fine-tuning that eliminates the need for floating-point operations in both inference and training.
Outcome:	The proposed framework reduces memory and compute costs while reducing memory usage.

Copied to clipboard

Xinxin Liu, Aaron Thomas, Cheng Zhang, Jianyi Cheng, Yiren Zhao, Xitong Gao

Astra: Activation-Space Tail-Eigenvector Low-Rank Adaptation of Large Language Models (2026.findings-acl)

Challenge:	Existing methods for fine-tuning large language models require expensive training on consumer-grade hardwares.
Approach:	They propose a sparsity-based approach that introduces trainable sparse adaptations to the weight matrices in the model and offers greater flexibility in selecting fine-tuned parameters.
Outcome:	The proposed method outperforms other methods for a simple yet effective baseline for nLP tasks while sacrificing performance.

Copied to clipboard

Kainan Liu, Yong Zhang, Ning Cheng, Yun Zhu, Yanmeng Wang, Shaojun Wang, Jing Xiao

HyperAdaLoRA: Accelerating LoRA Rank Allocation During Training via Hypernetworks without Sacrificing Performance (2026.findings-acl)

Challenge:	Existing methods for fine-tuning pre-trained models are limited due to suboptimal activation subspaces.
Approach:	They propose a method that leverages tail eigenvectors of model output activations to construct low-rank adapters.
Outcome:	The proposed method outperforms existing methods across 16 benchmarks and surpasses full fine-tuning in certain scenarios.

Copied to clipboard

Hao Zhang, Zhenjia Li, Yifan Gao, Xi Xiao, Heng Zhang, Shuyang Zhang, null Xiaoxincc, Bo Huang, Yuhang Wu, Tianyang Wang, Hao Xu

Small Data, Big Noise: Adversarial Training for Robust ParameterEfficient Fine-Tuning (2026.findings-acl)

Challenge:	Low-Rank Adaptation (LoRA) assumes a uniform rank r for each incremental matrix, not accounting for the varying significance of weight matrices across modules and layers.
Approach:	They propose a framework that allows for faster convergence of low-rank adaptive models . they use a hypernetwork to prune the outputs of the hypernetworks to generate parameters .
Outcome:	The proposed framework accelerates convergence of AdaLoRA by leveraging a hypernetwork.

Copied to clipboard

Eitan Cohen, Idan Simai, Uri Shaham

Look Within or Beyond? A Theoretical Comparison Between Parameter-Efficient and Full Fine-Tuning (2026.acl-long)

Challenge:	Parameter-Efficient Fine-Tuning (PEFT) is essential for adapting foundation models to downstream tasks, but current methods struggle with robustness to noise and performance degradation on limited training data.
Approach:	They propose a framework that brings adversarial training to PEFT to enhance model robustness and generalization, outperforming alternative approaches.
Outcome:	Experiments with two variants of the proposed framework show that it outperforms existing methods in low-resource settings and under word-level and character-level corruptions.

Copied to clipboard

YongKang Liu, Xingle Xu, Ercong Nie, Zijing Wang, Shi Feng, Daling Wang, Qian Li, Hinrich Schuetze